[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Gabriel39 · 2024-12-18T13:44:04Z

…ed tasks

What problem does this PR solve?

Now, a pipeline task which is canceled will be put in runnable queue and release memory once it ran. However, a runnable queue may have too much tasks which leads to a unacceptable delay for this canceled task. So this PR use a separate queue to process all canceled tasks.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2024-12-18T13:44:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2024-12-18T13:44:13Z

run buildall

github-actions · 2024-12-18T13:44:58Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-12-18T13:49:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-19T02:04:41Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T02:06:06Z

run buildall

github-actions · 2024-12-19T02:09:47Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-19T02:35:26Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10133/26066)
Line Coverage: 29.78% (85168/285986)
Region Coverage: 28.83% (43691/151539)
Branch Coverage: 25.37% (22197/87492)
Coverage Report: http://coverage.selectdb-in.cc/coverage/56171ee35e2ea084e2311b6db2ef278f5d6ec05d_56171ee35e2ea084e2311b6db2ef278f5d6ec05d/report/index.html

github-actions · 2024-12-19T02:59:27Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T03:02:28Z

run buildall

github-actions · 2024-12-19T03:02:39Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

be/src/pipeline/task_queue.cpp

doris-robot · 2024-12-19T03:33:52Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.78% (85183/285996)
Region Coverage: 28.84% (43698/151544)
Branch Coverage: 25.37% (22202/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/95971717ad6b94ae0e15d953ea17a8687f0a71bb_95971717ad6b94ae0e15d953ea17a8687f0a71bb/report/index.html

Gabriel39 · 2024-12-19T03:47:03Z

run buildall

github-actions · 2024-12-19T03:47:11Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

doris-robot · 2024-12-19T04:20:35Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.79% (85213/286000)
Region Coverage: 28.84% (43708/151547)
Branch Coverage: 25.37% (22200/87498)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e624dc5091fd94105e858c34d93632cdf06f4737_e624dc5091fd94105e858c34d93632cdf06f4737/report/index.html

Gabriel39 · 2024-12-19T07:24:33Z

run buildall

github-actions · 2024-12-19T07:24:57Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

doris-robot · 2024-12-19T08:12:31Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10133/26067)
Line Coverage: 29.78% (85171/285998)
Region Coverage: 28.83% (43691/151545)
Branch Coverage: 25.37% (22195/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42532ad7b23925dc5c5ec8051beca939e2ce4b0d_42532ad7b23925dc5c5ec8051beca939e2ce4b0d/report/index.html

github-actions · 2024-12-19T08:32:53Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T08:33:23Z

run buildall

doris-robot · 2024-12-19T09:02:37Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.78% (85173/285997)
Region Coverage: 28.83% (43692/151545)
Branch Coverage: 25.37% (22195/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6e2fe8b6325172cb7a79b2b3a6c829d20f225962_6e2fe8b6325172cb7a79b2b3a6c829d20f225962/report/index.html

be/src/pipeline/pipeline_task.cpp

be/src/pipeline/task_queue.h

be/src/pipeline/pipeline_task.h

be/src/pipeline/task_queue.cpp

github-actions · 2024-12-20T01:52:11Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-12-20T01:53:41Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

github-actions · 2024-12-20T01:56:40Z

be/src/pipeline/task_queue.cpp

@@ -172,7 +178,7 @@ PipelineTask* MultiCoreTaskQueue::take(int core_id) {
    return task;
 }

-PipelineTask* MultiCoreTaskQueue::_steal_take(int core_id) {
+std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {


warning: method '_steal_take' can be made const [readability-make-member-function-const]

Suggested change

std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {

std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) const {

be/src/pipeline/task_queue.h:126:

- std::shared_ptr<PipelineTask> _steal_take(int core_id); + std::shared_ptr<PipelineTask> _steal_take(int core_id) const;

Gabriel39 · 2024-12-20T01:57:43Z

run buildall

doris-robot · 2024-12-20T02:54:07Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26062)
Line Coverage: 29.80% (85212/285993)
Region Coverage: 28.91% (43479/150398)
Branch Coverage: 25.44% (22168/87154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1c6313a84d8dec05d6c3caef6e5b4d00c05a2d3_d1c6313a84d8dec05d6c3caef6e5b4d00c05a2d3/report/index.html

Gabriel39 · 2024-12-20T08:04:21Z

run buildall

doris-robot · 2024-12-20T08:32:50Z

TPC-H: Total hot run time: 39918 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

------ Round 1 ----------------------------------
q1	17806	7571	7279	7279
q2	2052	182	180	180
q3	10650	1170	1192	1170
q4	10322	728	715	715
q5	7589	2750	2721	2721
q6	236	149	155	149
q7	969	618	611	611
q8	9257	1894	1922	1894
q9	6675	6517	6407	6407
q10	7014	2300	2277	2277
q11	480	262	276	262
q12	419	215	223	215
q13	17793	2946	2958	2946
q14	247	227	205	205
q15	547	513	506	506
q16	652	606	579	579
q17	990	482	586	482
q18	7732	6660	6671	6660
q19	1370	965	916	916
q20	484	191	190	190
q21	4239	3243	3357	3243
q22	385	311	317	311
Total cold run time: 107908 ms
Total hot run time: 39918 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7277	7222	7244	7222
q2	332	232	231	231
q3	2946	2819	2840	2819
q4	2007	1744	1674	1674
q5	5360	5402	5458	5402
q6	221	137	140	137
q7	2173	1711	1688	1688
q8	3250	3415	3431	3415
q9	8715	8649	8632	8632
q10	3508	3479	3475	3475
q11	605	491	504	491
q12	772	581	550	550
q13	12011	2892	2975	2892
q14	296	268	260	260
q15	551	503	487	487
q16	678	624	649	624
q17	1801	1596	1563	1563
q18	7935	7328	7524	7328
q19	1716	1516	1534	1516
q20	2032	1865	1834	1834
q21	5528	5285	5272	5272
q22	654	589	547	547
Total cold run time: 70368 ms
Total hot run time: 58059 ms

doris-robot · 2024-12-20T08:42:08Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26062)
Line Coverage: 29.80% (85271/286101)
Region Coverage: 28.92% (43529/150496)
Branch Coverage: 25.44% (22169/87152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/08b123bfdd25c2b71e27e18afe2e396b80466034_08b123bfdd25c2b71e27e18afe2e396b80466034/report/index.html

doris-robot · 2024-12-20T08:44:56Z

TPC-DS: Total hot run time: 189933 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

query1	1000	402	404	402
query2	6528	2556	2391	2391
query3	6719	215	211	211
query4	33782	23350	23325	23325
query5	4391	469	456	456
query6	288	212	191	191
query7	4630	301	306	301
query8	313	244	229	229
query9	9692	2725	2716	2716
query10	487	256	249	249
query11	18006	15220	15061	15061
query12	163	106	105	105
query13	1694	414	412	412
query14	10837	6873	6914	6873
query15	307	191	179	179
query16	8307	460	482	460
query17	1754	583	606	583
query18	2148	317	318	317
query19	380	148	147	147
query20	119	117	115	115
query21	211	102	99	99
query22	4575	4408	4459	4408
query23	34546	33772	33192	33192
query24	11494	2514	2354	2354
query25	682	391	382	382
query26	1787	156	152	152
query27	2988	327	323	323
query28	7769	2419	2429	2419
query29	1045	403	408	403
query30	303	147	143	143
query31	1020	800	804	800
query32	101	59	58	58
query33	762	303	278	278
query34	994	527	517	517
query35	875	752	735	735
query36	1112	942	951	942
query37	282	80	75	75
query38	4124	4181	4018	4018
query39	1510	1408	1435	1408
query40	283	101	99	99
query41	47	43	48	43
query42	117	103	101	101
query43	542	520	499	499
query44	1257	796	795	795
query45	183	166	167	166
query46	1166	704	699	699
query47	1923	1815	1853	1815
query48	416	323	321	321
query49	1295	390	378	378
query50	794	395	395	395
query51	7229	7037	7124	7037
query52	102	95	90	90
query53	264	183	184	183
query54	1277	414	417	414
query55	89	82	77	77
query56	264	263	241	241
query57	1258	1141	1082	1082
query58	236	219	226	219
query59	3291	3362	3127	3127
query60	288	250	242	242
query61	108	111	111	111
query62	915	660	730	660
query63	218	196	189	189
query64	5068	674	681	674
query65	3252	3191	3227	3191
query66	1427	316	310	310
query67	15783	15673	15438	15438
query68	5835	550	556	550
query69	436	280	250	250
query70	1245	1114	1112	1112
query71	450	253	255	253
query72	6351	4123	4106	4106
query73	776	370	367	367
query74	10336	8868	8868	8868
query75	3470	2676	2689	2676
query76	3523	1187	1158	1158
query77	563	270	286	270
query78	11900	9998	9341	9341
query79	1111	596	595	595
query80	718	453	433	433
query81	543	228	233	228
query82	325	127	125	125
query83	262	213	142	142
query84	238	69	70	69
query85	776	307	299	299
query86	425	305	303	303
query87	4497	4472	4412	4412
query88	3729	2225	2205	2205
query89	406	294	294	294
query90	2249	197	194	194
query91	137	102	103	102
query92	66	55	53	53
query93	1291	543	547	543
query94	1026	283	297	283
query95	358	254	254	254
query96	626	281	280	280
query97	2838	2646	2663	2646
query98	214	203	197	197
query99	1508	1322	1300	1300
Total cold run time: 305045 ms
Total hot run time: 189933 ms

doris-robot · 2024-12-20T08:50:27Z

ClickBench: Total hot run time: 32.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.07	0.08
query4	1.60	0.10	0.11
query5	0.42	0.39	0.42
query6	1.16	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.51
query10	0.55	0.58	0.56
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	2.87	2.71	2.84
query15	0.89	0.82	0.82
query16	0.38	0.38	0.38
query17	1.06	1.06	1.00
query18	0.23	0.21	0.21
query19	1.95	1.78	1.98
query20	0.01	0.01	0.01
query21	15.35	0.58	0.56
query22	2.41	2.07	2.57
query23	17.11	1.01	0.71
query24	3.03	1.15	1.88
query25	0.17	0.23	0.05
query26	0.60	0.14	0.13
query27	0.04	0.05	0.05
query28	9.89	1.11	1.07
query29	12.57	3.22	3.22
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.25	0.47	0.46
query33	3.03	3.10	3.10
query34	17.16	4.48	4.55
query35	4.54	4.53	4.48
query36	0.68	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.03
query43	0.03	0.02	0.03
Total cold run time: 106.41 s
Total hot run time: 32.85 s

Gabriel39 · 2024-12-20T14:37:35Z

run buildall

doris-robot · 2024-12-20T15:11:48Z

TPC-H: Total hot run time: 39954 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

------ Round 1 ----------------------------------
q1	17594	7420	7289	7289
q2	2043	174	169	169
q3	10552	1145	1175	1145
q4	10234	730	730	730
q5	7615	2743	2620	2620
q6	246	154	155	154
q7	993	634	613	613
q8	9255	1826	1936	1826
q9	6577	6481	6388	6388
q10	6978	2281	2326	2281
q11	473	277	270	270
q12	429	217	227	217
q13	17782	2942	3017	2942
q14	262	209	208	208
q15	563	522	500	500
q16	659	590	597	590
q17	990	525	545	525
q18	7391	6778	6822	6778
q19	1343	942	947	942
q20	475	186	186	186
q21	4031	3279	3324	3279
q22	370	302	316	302
Total cold run time: 106855 ms
Total hot run time: 39954 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7210	7238	7210
q2	334	231	231	231
q3	2894	2741	2970	2741
q4	2092	1860	1849	1849
q5	5571	5680	5674	5674
q6	222	144	144	144
q7	2212	1792	1906	1792
q8	3395	3469	3497	3469
q9	8879	8935	8874	8874
q10	3638	3595	3544	3544
q11	599	507	510	507
q12	793	639	597	597
q13	12873	3108	3037	3037
q14	299	302	266	266
q15	584	499	512	499
q16	685	642	651	642
q17	1865	1637	1616	1616
q18	7815	7535	7318	7318
q19	1683	1474	1551	1474
q20	2073	1822	1888	1822
q21	5418	5256	5298	5256
q22	662	564	574	564
Total cold run time: 71884 ms
Total hot run time: 59126 ms

doris-robot · 2024-12-20T15:23:51Z

TPC-DS: Total hot run time: 189600 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

query1	969	389	391	389
query2	6538	2375	2292	2292
query3	6705	229	220	220
query4	33636	23449	23376	23376
query5	4375	453	461	453
query6	282	186	187	186
query7	4622	311	313	311
query8	308	236	240	236
query9	9353	2697	2689	2689
query10	471	256	253	253
query11	17985	15068	15302	15068
query12	165	106	106	106
query13	1764	425	433	425
query14	11869	6769	7135	6769
query15	325	190	194	190
query16	8252	450	401	401
query17	1760	606	578	578
query18	2123	285	289	285
query19	365	167	148	148
query20	128	109	109	109
query21	211	103	103	103
query22	4767	4513	4184	4184
query23	34787	33391	33420	33391
query24	11015	2482	2436	2436
query25	681	376	388	376
query26	1859	159	158	158
query27	2904	338	320	320
query28	7942	2437	2434	2434
query29	1041	404	416	404
query30	304	148	152	148
query31	1038	826	812	812
query32	94	57	60	57
query33	778	295	286	286
query34	1006	508	528	508
query35	898	754	729	729
query36	1100	932	933	932
query37	271	76	75	75
query38	4234	4058	4110	4058
query39	1505	1418	1412	1412
query40	300	101	100	100
query41	47	45	50	45
query42	130	101	106	101
query43	546	511	490	490
query44	1281	805	813	805
query45	184	167	162	162
query46	1171	716	718	716
query47	1945	1850	1828	1828
query48	409	316	327	316
query49	1278	383	399	383
query50	794	380	385	380
query51	7119	7198	7131	7131
query52	104	92	95	92
query53	255	185	187	185
query54	1195	400	408	400
query55	78	82	83	82
query56	268	246	248	246
query57	1265	1098	1129	1098
query58	240	218	228	218
query59	3173	2988	2924	2924
query60	296	254	255	254
query61	112	110	117	110
query62	885	674	690	674
query63	217	190	186	186
query64	5043	677	681	677
query65	3272	3216	3243	3216
query66	1419	312	303	303
query67	15964	15556	15610	15556
query68	5736	564	561	561
query69	427	255	257	255
query70	1236	1048	1153	1048
query71	449	255	257	255
query72	6439	4036	4035	4035
query73	780	351	361	351
query74	9272	8881	8778	8778
query75	3411	2721	2654	2654
query76	3602	1088	1054	1054
query77	551	286	274	274
query78	10240	9450	9386	9386
query79	1680	611	590	590
query80	1043	481	417	417
query81	527	233	235	233
query82	923	115	121	115
query83	244	163	145	145
query84	235	70	72	70
query85	1264	302	313	302
query86	394	302	296	296
query87	4420	4419	4438	4419
query88	3348	2236	2199	2199
query89	424	297	293	293
query90	2167	197	192	192
query91	134	105	114	105
query92	62	49	59	49
query93	1282	535	545	535
query94	1151	285	298	285
query95	357	256	252	252
query96	611	277	287	277
query97	2805	2692	2688	2688
query98	223	205	196	196
query99	1570	1306	1320	1306
Total cold run time: 304427 ms
Total hot run time: 189600 ms

doris-robot · 2024-12-20T15:29:22Z

ClickBench: Total hot run time: 32.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.44	0.41	0.40
query6	1.16	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.15	0.10	0.11
query12	0.13	0.11	0.11
query13	0.61	0.62	0.59
query14	2.84	2.86	2.75
query15	0.89	0.83	0.82
query16	0.41	0.39	0.37
query17	1.03	1.05	1.01
query18	0.22	0.21	0.21
query19	1.87	1.78	2.02
query20	0.02	0.00	0.01
query21	15.35	0.61	0.60
query22	3.08	2.40	1.35
query23	17.00	1.04	1.01
query24	3.42	0.81	1.44
query25	0.32	0.06	0.19
query26	0.47	0.15	0.14
query27	0.06	0.05	0.04
query28	10.32	1.11	1.07
query29	12.60	3.22	3.23
query30	0.24	0.06	0.06
query31	2.84	0.38	0.40
query32	3.24	0.48	0.47
query33	3.11	3.24	3.15
query34	16.83	4.45	4.50
query35	4.51	4.53	4.48
query36	0.69	0.50	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.47 s
Total hot run time: 32.22 s

doris-robot · 2024-12-20T15:41:47Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10133/26064)
Line Coverage: 29.81% (85300/286148)
Region Coverage: 28.93% (43553/150530)
Branch Coverage: 25.44% (22181/87178)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8011705f5621ae333b9a538477e31119ed66c2ab_8011705f5621ae333b9a538477e31119ed66c2ab/report/index.html

be/src/pipeline/task_scheduler.cpp

…ed tasks

Gabriel39 · 2024-12-30T15:14:13Z

run buildall

doris-robot · 2024-12-30T16:02:16Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10121/26026)
Line Coverage: 29.87% (85499/286193)
Region Coverage: 29.01% (43709/150681)
Branch Coverage: 25.54% (22297/87306)
Coverage Report: http://coverage.selectdb-in.cc/coverage/592a371179b3444929f2a66ccdb1558eb0d41861_592a371179b3444929f2a66ccdb1558eb0d41861/report/index.html

yiguolei · 2024-12-31T02:37:30Z

be/src/pipeline/task_scheduler.cpp

+            auto expected = TaskState::VALID;
+            if (!holder->state.compare_exchange_strong(expected, TaskState::RUNNING)) {
+                if (expected == TaskState::RUNNING) {
+                    static_cast<void>(_task_queue.push_back(holder, index));


如果状态already 是running了，说明他已经在另外一个队列里被执行了，应该由另外一个队列往里放。

不是的，这里是为了防止一个task重复进入调度队列，这里已经放进了预期执行的队列里，只是要等这个task在前一个执行线程退出。

Gabriel39 · 2024-12-31T03:23:52Z

run buildall

Gabriel39 · 2024-12-31T03:39:08Z

run buildall

doris-robot · 2024-12-31T04:07:54Z

TPC-H: Total hot run time: 32585 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f6c7f622d28669979d2a58bcad05339c5890d19, data reload: false

------ Round 1 ----------------------------------
q1	17576	6178	6004	6004
q2	2045	302	182	182
q3	10420	1239	721	721
q4	10218	867	431	431
q5	7756	2187	1987	1987
q6	207	190	146	146
q7	892	768	598	598
q8	9264	1372	1170	1170
q9	5359	4912	4953	4912
q10	6764	2301	1861	1861
q11	469	277	257	257
q12	348	374	214	214
q13	17764	3542	2913	2913
q14	231	239	212	212
q15	558	498	507	498
q16	645	602	590	590
q17	591	849	326	326
q18	6963	6524	6574	6524
q19	2680	974	587	587
q20	310	312	186	186
q21	2897	2223	1951	1951
q22	360	338	315	315
Total cold run time: 104317 ms
Total hot run time: 32585 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6323	6227	6220	6220
q2	236	337	226	226
q3	2270	2673	2306	2306
q4	1397	1839	1336	1336
q5	4310	4761	4831	4761
q6	186	179	141	141
q7	2110	1982	1847	1847
q8	2662	2795	2671	2671
q9	7274	7318	7295	7295
q10	3071	3354	2731	2731
q11	575	500	494	494
q12	676	807	606	606
q13	3332	3793	3114	3114
q14	285	332	275	275
q15	581	517	511	511
q16	650	677	652	652
q17	1205	1747	1238	1238
q18	7618	7626	7441	7441
q19	848	1130	1087	1087
q20	2016	2021	1971	1971
q21	5822	5140	4908	4908
q22	616	657	598	598
Total cold run time: 54063 ms
Total hot run time: 52429 ms

doris-robot · 2024-12-31T04:19:38Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10122/26024)
Line Coverage: 29.88% (85510/286200)
Region Coverage: 29.01% (43715/150687)
Branch Coverage: 25.55% (22304/87310)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2f6c7f622d28669979d2a58bcad05339c5890d19_2f6c7f622d28669979d2a58bcad05339c5890d19/report/index.html

doris-robot · 2024-12-31T04:20:21Z

TPC-DS: Total hot run time: 196680 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f6c7f622d28669979d2a58bcad05339c5890d19, data reload: false

query1	1306	928	993	928
query2	6483	2337	2347	2337
query3	10990	4786	4712	4712
query4	33078	23902	23507	23507
query5	4326	634	455	455
query6	268	198	178	178
query7	3972	502	312	312
query8	315	250	221	221
query9	9423	2744	2715	2715
query10	467	311	259	259
query11	18106	15445	15131	15131
query12	161	112	112	112
query13	1584	572	426	426
query14	9398	7322	6865	6865
query15	258	220	190	190
query16	7397	646	467	467
query17	1626	765	586	586
query18	1938	416	345	345
query19	226	196	166	166
query20	122	114	130	114
query21	209	129	117	117
query22	4755	4801	4465	4465
query23	34372	33441	33631	33441
query24	6808	2343	2359	2343
query25	492	464	417	417
query26	1006	274	156	156
query27	2313	467	330	330
query28	5697	2511	2492	2492
query29	643	551	432	432
query30	205	186	154	154
query31	996	904	843	843
query32	75	57	60	57
query33	491	368	299	299
query34	764	862	535	535
query35	804	855	782	782
query36	1026	1075	998	998
query37	119	109	76	76
query38	4301	4181	4247	4181
query39	1503	1486	1449	1449
query40	208	119	102	102
query41	48	46	69	46
query42	126	105	104	104
query43	524	536	506	506
query44	1315	845	823	823
query45	186	178	172	172
query46	889	1080	668	668
query47	2047	1997	1947	1947
query48	399	407	321	321
query49	739	503	397	397
query50	679	670	404	404
query51	7197	7366	7269	7269
query52	102	101	89	89
query53	227	260	184	184
query54	497	522	409	409
query55	87	84	83	83
query56	266	258	252	252
query57	1277	1214	1160	1160
query58	244	223	230	223
query59	3193	3260	3179	3179
query60	283	261	253	253
query61	138	124	125	124
query62	874	810	762	762
query63	227	204	209	204
query64	4075	1035	668	668
query65	3352	3231	3278	3231
query66	966	427	303	303
query67	16619	16001	15515	15515
query68	9826	765	527	527
query69	503	300	256	256
query70	1214	1155	1134	1134
query71	450	299	252	252
query72	6252	3855	3927	3855
query73	1350	756	361	361
query74	10073	9039	9077	9039
query75	4604	3115	2655	2655
query76	5637	1201	793	793
query77	1032	354	294	294
query78	10089	10222	9342	9342
query79	5587	900	586	586
query80	689	526	427	427
query81	497	270	238	238
query82	236	158	136	136
query83	203	257	163	163
query84	292	91	72	72
query85	747	360	295	295
query86	352	316	310	310
query87	4564	4450	4365	4365
query88	4064	2231	2213	2213
query89	442	324	291	291
query90	2149	187	186	186
query91	143	140	106	106
query92	61	56	54	54
query93	3624	902	545	545
query94	678	395	285	285
query95	333	270	296	270
query96	490	611	283	283
query97	2684	2851	2703	2703
query98	231	208	200	200
query99	1649	1580	1453	1453
Total cold run time: 305552 ms
Total hot run time: 196680 ms

doris-robot · 2024-12-31T04:25:41Z

ClickBench: Total hot run time: 31.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f6c7f622d28669979d2a58bcad05339c5890d19, data reload: false

query1	0.03	0.05	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.43	0.42	0.40
query6	1.19	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.59	0.50	0.52
query10	0.56	0.58	0.56
query11	0.15	0.11	0.10
query12	0.15	0.11	0.10
query13	0.61	0.60	0.60
query14	2.73	2.77	2.85
query15	0.89	0.83	0.82
query16	0.38	0.38	0.38
query17	1.06	1.01	1.07
query18	0.23	0.22	0.20
query19	1.96	1.81	1.99
query20	0.02	0.01	0.01
query21	15.36	0.90	0.57
query22	0.74	0.73	0.69
query23	15.33	1.43	0.56
query24	3.07	1.19	1.21
query25	0.23	0.18	0.07
query26	0.25	0.14	0.13
query27	0.07	0.04	0.05
query28	14.05	1.52	1.06
query29	12.54	4.01	3.29
query30	0.25	0.09	0.06
query31	2.85	0.60	0.39
query32	3.24	0.55	0.45
query33	3.13	3.10	3.10
query34	16.97	5.09	4.50
query35	4.45	4.45	4.46
query36	0.65	0.50	0.50
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.14	0.12
query41	0.07	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.64 s
Total hot run time: 31.52 s

Gabriel39 force-pushed the imp_1218 branch from f4fda83 to 9597171 Compare December 19, 2024 03:01

github-actions bot reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.cpp Outdated Show resolved Hide resolved

be/src/pipeline/task_queue.cpp Outdated Show resolved Hide resolved

Gabriel39 force-pushed the imp_1218 branch from e624dc5 to 42532ad Compare December 19, 2024 07:24

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/pipeline_task.cpp Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/pipeline_task.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.cpp Outdated Show resolved Hide resolved

github-actions bot reviewed Dec 20, 2024

View reviewed changes

yiguolei reviewed Dec 30, 2024

View reviewed changes

be/src/pipeline/task_scheduler.cpp Show resolved Hide resolved

yiguolei reviewed Dec 30, 2024

View reviewed changes

be/src/pipeline/task_scheduler.cpp Show resolved Hide resolved

[Improvement](scheduler) Use a separate eager queue to execute cancel…

592a371

…ed tasks

Gabriel39 force-pushed the imp_1218 branch from 50bb1d5 to 592a371 Compare December 30, 2024 15:12

yiguolei reviewed Dec 31, 2024

View reviewed changes

update

8c8388f

udpate

2f6c7f6

	std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {
	std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) const {

[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Are you sure you want to change the base?

[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Conversation

Gabriel39 commented Dec 18, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Dec 18, 2024

Gabriel39 commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Dec 19, 2024

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

doris-robot commented Dec 19, 2024

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 20, 2024

sh-checker report

github-actions bot commented Dec 20, 2024

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Dec 20, 2024

Choose a reason for hiding this comment

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 30, 2024

doris-robot commented Dec 30, 2024

yiguolei Dec 31, 2024

Choose a reason for hiding this comment

Gabriel39 Dec 31, 2024

Choose a reason for hiding this comment

Gabriel39 commented Dec 31, 2024

Gabriel39 commented Dec 31, 2024

doris-robot commented Dec 31, 2024

doris-robot commented Dec 31, 2024

doris-robot commented Dec 31, 2024

doris-robot commented Dec 31, 2024

Gabriel39 commented Dec 18, 2024 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`